The present invention relates to a method of processing the color component signal (image signal) obtained from a solid-state image pickup device, an image signal processing apparatus, an image signal generating apparatus comprising the image signal processing apparatus and the solid-state image pickup device, and a computer program product for the image signal processing method.
The solid-state image pickup device or the image sensor has a two-dimensional arrangement of a plurality of photoelectric elements such as photodiodes. For example, both the three primary color signals such as R, B and G signals and the luminance signal are produced in a single-chip solid-state image pickup device. A monochromatic color filter of R (red light penetrating), G (green light penetrating) or B (blue light penetrating) is arranged on the light-receiving surface of each photoelectric element. A white filter (a filter of white light penetrating) is arranged in place of the G filter in some cases. A plurality of photoelectric elements including these three types of color filters are arranged on the light-receiving surface in a predetermined pattern such as Bayer arrangement pattern. Hereinafter, we will explain examples wherein each photoelectric element corresponds to a pixel. The solid-state image pickup device is of either CCD type or MOS FET type.
Generally, the color component signals composed of all the three primary colors are required from each pixel for producing high quality full-color image signals. In the single-chip solid-state image pickup device, however, only a monochromatic color component signal corresponding to the color of the filter of a respective photoelectric element of a pixel can be produced in each pixel. In each pixel of a given color in the single-chip solid-state image pickup device the two remaining color component signals are produced in such a manner that the signals from the adjacent pixels having different color filters are subjected to the signal processing such as interpolation.
An example of the image signal processing method as described above is disclosed in U.S. Pat. No. 5,382,976. This US patent describes a signal processing method for a single image sensor including a red pixel, a green pixel and a blue pixel. For the red pixel or the blue pixel not including the green component, the green value is obtained by interpolating the signal values of the pixels adjacent to the particular pixel. Also, the US patent includes the description to the effect that with regard to the green value of phase “00” containing the green value or phase “1 μl”, the adjacent pixel values are not interpolated but the signal values of the phases are used as they are without interpolation as indicated by G=G(0,0) in FIGS. 4A and 4B of the U.S. patent.
Now, a specific example of an image signal processing method will be explained with reference to the drawings.
FIG. 2 is a diagram showing an example a block configuration of an image signal processing apparatus comprising a single-chip solid-state image pickup device and various processing circuits for processing the image signal picked up by the solid-state image pickup device. In FIG. 2, reference numeral 1 designates a solid-state image pickup device such as a charge-coupled device (CCD) image sensor for converting the imaging light into electric charge and outputting it as a video signal, numeral 2 a sample-hold and automatic gain control (CDS&AGC) circuit (CDS: Correlated Double Sampling) for sample holding the video signal outputted from the CCD 1 and outputting the video signal by amplifying the sample hold signal to a required level, and numeral 3 an A/D converter for converting the analog video signal from the CDS&AGC circuit 2 into a digital video signal.
Numeral 4 designates a DSP (digital signal processor) circuit for processing the video signal from the A/D converter 3 as required.
The color filters of the solid-state image pickup device (CCD) 1 are arranged in Bayer arrangement pattern. FIG. 3A shows the Bayer arrangement pattern and the arrangement of the pixel addresses attached to the pixels. In FIG. 3A, a green light transmitting filter (a filter of green light penetrating) is designated as G, a red light transmitting filter (a filter of red light penetrating) as R and a blue light transmitting filter (a filter of blue light penetrating) as B. Further, the number 1 is attached to the highest row and the leftmost column. For example, the pixel address of a pixel having a green light transmitting filter located on the mth row from top and the nth column from left is expressed as Gm,n. The symbol of the pixel address is also expressed as the value of the color component signal obtained by picking up an image in the particular pixel.
A method of image signal processing for the green component signal constituting one of the processes executed in the DSP circuit 4 described above will be explained with reference to FIG. 4C showing an example of the pixel interpolation method. The method (i) or (ii) described below is adaptively used for each pixel address. Specifically, in method (i), as for the green component signal level of a pixel address (e.g. G22) having a green light transmitting filter, the value of the green component signal level of the particular pixel address is used as it is. In method (ii), on the other hand, as for the green component signal level of a pixel address (e.g. R23) having a red light transmitting filter or the green component signal level of a pixel address (e.g. B34) having a blue light transmitting filter, the average value of the green component signal levels obtained from the pixels (e.g. G13, G22, G24, G33) having a green light transmitting filter at the upper, lower, left and right adjacent pixel addresses is used.
A specific example of the signal processing method (ii) described above will be explained in more detail. As shown in FIG. 4C, in the case where the image signal is processed for the pixel address R23 to generate the green component signal level g23 by interpolation, for example, the value is calculated fromg23=(G13+G22+G24+G33)/4   (1) 
Another method of calculation included in the signal processing method (ii) uses the values of only the left and right adjacent pixels of the pixel address R23 as shown in equation (2) below for calculation.g23=(G22+G24)/2  (2) Various other alternative methods are conceivable including a method in which only the upper and lower adjacent pixel values or only the diagonally adjacent pixel values of the pixel address R23 are used for calculation.
FIG. 4A is a diagram showing an interpolation method for the R component signal at the position of the pixel address G22. In this case, the red component signal at the position of G22 has a pixel level equal to the average of the imaging red component signal levels of the pixel address R21 and the pixel address R23. FIG. 4B is a diagram showing an interpolation method for the B component signal at the position of the pixel address G22. In this case, the blue component signal at the position of G22 has a pixel level equal to the average of the imaging blue component signal levels of the pixel address B12 and the pixel address B32. Further, FIG. 4D is a diagram showing an interpolation method for the B component signal at the position of the pixel address R23. In this case, the blue component signal at the position of R23 has a pixel level equal to the average of the imaging blue component signal levels at the respective positions of the pixel addresses B12, B14, B32 and B34.
The signal processing method described above has the following problem. Specifically, assume that the band limiting range of the source follower circuit of the amplifier 15 of the output unit of the CCD used for securing the required S/N ratio, the bandwidth of the CDS circuit 2 or the band limiting range of the process amplifier is excessively narrow. Then, the waveform of the output signal is distorted in each case. As a result, a horizontal line having the blue light transmitting filter such as the (m−1)th line (row) in FIG. 3A and the horizontal line having the red light transmitting filter such as the mth line (row) in FIG. 3A have different waveform distortion amounts of the green component signal. Thus, the error levels applied to the green component signals are different between adjacent lines, often resulting in variations of the green component signal level, even when a uniform green component light in the imaging light is applied from the object to each pixel. The green component signals varied from one line to another causes horizontal stripes of noises on the image subjected to the signal processing as shown in FIG. 4C.
Also, in the case where the conditions for the production process undergo a change, the filter characteristic of the green light transmitting filter among the CCD color filters may be varied in every other horizontal scanning line, i.e. between a horizontal line having the blue light transmitting filter and a horizontal line having the red light transmitting filter, which often causes variations of the green component signal level between different lines.
This often leads to the problem that even when a uniform green component light in the imaging light is applied from an object to each pixel, the levels of the green component signals corresponding to the pixels of the green light transmitting filters outputted from the CCD may be varied between even-row horizontal lines and odd-row horizontal lines.
FIGS. 6A and 6B show waveforms obtained by sampling the output signal of the CCD in a correlated double sampling (CDS) circuit when the CCD having the aforementioned problem picks up an object having a uniform brightness and color in a whole picture frame. FIG. 6A shows a signal waveform of a horizontal line (the mth horizontal line in FIG. 3A, for example) having a red light transmitting filter and green one, and FIG. 6B shows a signal waveform of a horizontal line (the (m−1)th horizontal line in FIG. 3A, for example) having a blue light transmitting filter and green one.
In FIGS. 6A and 6B, numerals 100, 200 designate reference voltage signal (for example, an optical black signal) waveforms. Numeral 110 designates an image signal waveform corresponding to the reference voltage signal waveform 100, and numeral 210 designates an image signal waveform corresponding to the reference voltage signal waveform 200. The values of the color component signals of the image signal waveforms 110, 210 are indicated by the level difference with the reference voltage signal waveforms 100, 200, respectively. In the waveform 110, from left to right, the level waveforms of the red component signal, the green component signal and the red component signal are shown, while in the waveform 210, from left to right, the level waveforms of the green component signal, the blue component signal and the green component signal are shown.
Comparison between the green component signal of the waveform 110 in FIG. 6A and the green component signal of the waveform 210 in FIG. 6B shows that the signal levels are different between them due to the problem described above. In the case where an image is displayed on a monitor (not shown) inserted in a subsequent stage, this level difference is seen to cause a level difference between horizontal lines even in a displayed image produced by imaging an object uniformly. Thus the image is displayed in horizontal stripes or speckles, thereby extremely deteriorating the image quality.